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Overview Nilo 
| University 

= Multi-scale metric & feedback loops 
e Design hazard analysis 


e Operational risk mitigation 
e Lifecycle discovery of surprises 
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= Safety Performance Indicators (SPIs) WO « 
e Beyond ‘vehicle acted unsafely” us ) ee. 
e Beyond real-time dynamic risk measurement 





e It's all about monitoring safety case validity 
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Traditional Hazard Analysis Nion 

= Risk Analysis (e.g., start with HARA) 

e List all applicable hazards 

e Characterize the resultant risk 

e Mitigate risk as needed Pee rap 

e Document all risks acceptably mitigated 
m Use various techniques to create hazard list 

e Lessons learned (previous projects; industry) 

e Brainstorming & analysis techniques 

—- HAZOP, STPA, .... bring your own favorite approach ... 

= Limitation: unknown hazards 

e But, human is responsible for overall system safety 
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Hazard Analysis for ADAS Non 
= Operating in the open world 
e All hazards arent known 
e New hazards will appear 
= Safety of the Intended Function (SOTIF) 
e Operate in the real world HAZARD 
e Observe “triggering events” ey 
e Mitigate discovered hazards 
e Repeat 
= Limitation: unseen triggering events 
e But, human is responsible for system safety 













SOTIF 
TRIGGERING 
EVENTS 
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Carnegie 


Pre-Autonomy & ADAS Feedback Models tii.,., 


= Driver does dynamic risk mitigation 
= Recalls for technical faults 
e Recalls are never supposed to happen 


DRIVER 
HAZARD | - > 
ANALYSIS { \ cin aa lg et | 
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| SOTIF | 
TRIGGERING 
EVENTS 


Carnegie 
Hazard Analysis for Full Autonomy Uersity 
= Still an open world with unknowns & changes 
e But ... 70 Auman driver responsible 


= Use Positive Trust Balance TRUSTWORTHY POSITIVE RISK BALANCE 
e Engineering rigor ema Dyan e Dyan Dame 
e Practicable validation 
e Strong safety culture 
me andr: 
e Field feedback 
to handle surprises 


BUILD IT RIGHT 
TEST IT RIGHT 


( IMPROVE IT RIGHT 


LIVE IT RIGHT 














Engineering Validation Feedback 
Rigor Culture 


= Good fit to UL 4600 = Safety Cases © 2022 Philip Koopman 6 


Carnegie 


Safety Arguments (Safety Case) ype Fe 


= Claim — a property of the system 


e “System avoids pedestrians” CLAIM 


= Argument —- why this is true 


e “Detect & maneuver to avoid” 
oS fa) ee @ 


m Evidence — supports argument 
e Tests, analysis, simulations, ... 


= Sub-claims/arguments address 
complexity 


e “Detects pedestrians” // evidence 
e “Maneuvers around detected pedestrians’ // evidence 
e “Stops if can't maneuver’ // evidence 
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Carnegie 
Default SDC Feedback Model ieee! 
= Safety Case argues acceptable risk — without driver 
e Perhaps Positive Risk Balance (“safer than human”) 
e Update in response to incidents and loss events 













~ SOTIF 


HAZARD | 
ANALYSIS TRIGGERING ‘Leia 
EVENTS 


LOSS EVENTS 


e But, deployment only yields lagging metrics 
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Carnegie 


Safety Performance Indicators (SPIs) —tkiw".,, 


= SPis monitor the validity of safety ci case claims 


LAGGING Vehicle is Safe ~ 
METRICS 





















| —~" CLAIMS-ONLY 
- & 7 VIEW OF 

SAFETY CASE 
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-: ——— Le 
LEADING lm ms Sensor Cleaning 


METRICS 


Data Fusion _ Data Fusion Effective ¢ > 
SW Quality Test err ee 


SP 























© 2022 Philip Koopman 9 





Examples of SPlis Nien 

= “Acts dangerously’ is only one dimension of SPls 

e Violation rate of pedestrian buffer zones 

e Time spent too close per RSS following distance 
= Components meet safety related requirements 

e False negative/positive detection rates 

e Correlated multi-sensor failure rates 
= Design & Lifecycle considerations 

e Design process quality defect rates 

e Maintenance & inspection defect rates 
= Is it relevant to safety? =} Safety Case =} SPlis 
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Carnegie 


KPI vs. SPI Contrast _ University 


=m Distance to object: 
e KPI: average and variance of clearance 
e SPI: how often SDC violates safe clearance limit 
= Sensor effectiveness: 
e KPI: detection rate, SNR per sensor 
e SPI: concurrent multi-sensor detection failure 
e SPI: loss of calibration 
= Pedestrian perception: 
e KPI: accuracy, precision, recall 
e SPI: false negative more than <k> consecutive frames 
e SPI: systematic under-performance on sub-classes 
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, - F J ‘ Carnegie 
_ Runtime Monitoring Implications Unversity 
= Responsibility-Sensitive Safety (RSS) Scenario: 


» Pomcaces! 
__“*max,brake 2 << V. Amin, brake 2 
=| : So 
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e Safety monitor: increase distance if too close in case of panic stop 
e KPI: best effort separation given driving conditions 
e SPls: situation more dangerous than expected (e.g., ODD issues) 

— Spent more time in too-dense traffic than expected 

— Lead/own vehicle brake violate expectations 


— Other vehicles panic brake more often than assumed ~ 
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Carnegie 


SPlis and Lifecycle Feedback LCi 


= SPI measures validity of a safety case claim 
= a SPI value violation means safety case is invalid 


= Root cause analysis might reveal: 
e Design process execution defect \ \\\i 1 My / 
e Design defect \\ \ / “ 
e Hazard analysis gap \ 
e SOTIF analysis gap =\ SP! 
e Training data bias 
e Evidence gap, or defect 
e Assumption error 





© 2022 Philip Koopman 13 


Carnegie 


_ SPI-Based Feedback Approach ee that 


= Safety Case argues acceptable risk 
e SPIis monitor validity of safety case 


SOTIF 


HAZARD TRIGGERING EVENTS 


ANALYSIS 7 —_ 


RUN-TIME __ 
SAFETY / — 
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Carnegie 
ellon 
University 


Monitoring incidents is only part of feedback 


Removing human means mitigating surprise | 
e Tactical: run-time safety monitoring | 
e Strategic: run-time SPI monitoring 


SPlis provide feedback on: 

e Design quality & process maturity 
e Testing coverage 

e Lifecycle procedure execution 


eee) ey a eS * aA 
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